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ABSTRACT 

Inspired from non-equilibrium statistical physics models, a 
general framework enabling the definition and synthesis of 
stationary time series with a priori prescribed and controlled 
joint distributions is constructed. Its central feature consists of 
preserving for the joint distribution the simple product struc- 
ture it has under independence while enabling to input con- 
trolled and prescribed dependencies amongst samples. To that 
end, it is based on products of d-dimensional matrices, whose 
entries consist of valid distributions. The statistical properties 
of the thus defined time series are studied in details. Having 
been able to recast this framework into that of Hidden Markov 
Models enabled us to obtain an efficient synthesis procedure. 
Pedagogical well-chosen examples (time series with the same 
marginal distribution, same covariance function, but different 
joint distributions) aim at illustrating the power and potential 
of the approach and at showing how targeted statistical prop- 
erties can be actually prescribed. 

Index Terms — Time Series Synthesis, Joint Distribution, 
A priori Prescription, Hidden Markov Model 

1. INTRODUCTION 

In modern signal processing, the need to produce numerically 
random vectors whose joint distributions are fully prescribed 
and controlled is more and more common. Typical examples 
are Bayesian estimation (with Monte Carlo Markov Chain 
or variational based resolution schemes, cf. e.g., HI) or esti- 
mation performance assessment (where estimators are bench- 
marked on time series with prescribed distributions). For uni- 
variate Gaussian stationary time series, the so-called Circu- 
lant Embedded Matrix synthesis procedure is considered 
as the state of the art solution. For non Gaussian station- 
ary time series, various approaches were proposed that aim 
at controlling both the marginal distribution and the covari- 
ance structure of the process (cf. e.g., [3, 4| and references 
therein for reviews). Often, such methods suffer from two 
major limitations: The joint distribution cannot be prescribed 
a priori and is hence not controlled and results as a conse- 
quence of the details of the synthesis procedure; Distributions 
that consist of mixtures of elementary laws cannot always eas- 



ily be combined with prescribed covariance. The very general 
framework of Markov Chain simulation offers an alternative 
and broad class of solutions, focusing on the modeling of lo- 
cal dynamical properties, while not explicitly putting the em- 
phasis on a direct prescription of the joint distributions of the 
process. 

Instead, in the present contribution, the focus is on the 
synthesis of stationary time series whose joint distributions 
are explicitly prescribed and chosen a priori. It is based 
on a construction inspired from out-of-equilibrium statis- 
tical physics models (cf. e.g., ||5] |6] 13). It is founded 
on the central idea that the joint distribution of a vector 
X_ N = xi, X2, ■ ■ ■ , %n is still written as a (oriented) product: 
Px <x Ilfc=i Rd(xk), where, however the Rd& are not single 
distributions, but instead <i-dimensional matrices of distribu- 
tions. The key results of the present contribution consist of 
first deriving the general properties of this construction and 
then to specify it to a very interesting case specifically suited 
for stationary time series with all joint distributions prescribed 
(cf. Section 2). An efficient synthesis procedure is devised 
by explicitly recasting this framework into that of Hidden 
Markov Models (cf. Section 3). Finally, explicit examples 
are studied and simulated numerically, aiming at illustrating 
the power of the proposed approach: They consist of time 
series sharing the same complicated (mixture) marginal dis- 
tribution, the same covariance functions, but different joint 
distributions (cf. Section 4). 

2. JOINT DISTRIBUTIONS AS MATRIX PRODUCT 

General Framework. Let us first define a general formal- 
ism for the construction of joint distribution functions of ran- 
dom vectors 2Gv> inspired from non equilibrium statistical 
physics models (cf. e.g. [5, 6 7|) and based on products of 
matrices of distributions. Let Rd(x) denote a d-dimensional 
matrix with entries 

Rd{x)ij = £i.jV(x) ij , (1) 

where T'(x)ij are valid distribution functions and £ij arbi- 
trary positive numbers, forming the matrix £. Let A denote 
an arbitrary, but fixed strictly positive and non-random, ma- 



trix and C(M) the linear form applied to matrix M defined as 
C(M) =tr(A T M). 

Further, let X = {AT„}i<„<Ar denote a random vector, of 
chosen size N, explicitly defined via its joint distribution: 



_ £ (ilk M**) 



(2) 



where ]J k Rd{x k ) = Rd(%i) ■ ■ • -Rd(^iv) denotes the ori- 
ented product (i.e., the order of the factors is fixed and can- 
not be changed). It is straightforward to check that the joint 
choice of strictly positive entries for matrices A and £ , and of 
valid distribution functions for V(x)i j is sufficient to ensure 
that Eq. (|2| defines a valid joint distribution function. 

From these definitions, calculations and matrix manipu- 
lations not reported here enabled us to derive a number of 
statistical properties of the vector {AT„}i< n <jv- Its univari- 
ate (marginal) distributions and one-sample moments take ex- 
plicit forms (with M (q) — J R x q Rd(x)dx): 
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Furthermore, the joint p-sample moments read (with p € N, 
ki < ■ ■ ■ < k p and q r the order associated to the entry x k )'■ 

C (S*- 1 (nr 1 M(g r )£*H-i-*r-i) M(q s )£ N - k < 
C{£ N ) 

(5) 

Stationary time series. Let us now focus on specific 
choices for matrices A and E, of interest here to construct 
stationary time series: 



Aij = i £ = aI d + (iJ dl with a + /3 = 1, (6) 



Id is the d-dimensional unity matrix and Jd £ Md(M.) defined 
as: 
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Such choices for A and £ enabled us to show (full calcu- 
lations not reported here) that: 
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This further enables us to obtain that the results above can be 
specified as: 
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E[X«]=£(M(g)), 
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These relations clearly indicate that the vector {X n }i<„<N 
can now be regarded as a stationary time series: All joint 
statistics depend only on time differences, k r +i — k r . Both 
its marginal and joint probabilities are prescribed by the 
choices of £ and of the Vij. Also, Eq. (Hi constitutes a key 



result with respect to applications as it clearly shows that the 
joint statistics of order q of the time series can be prescribed 
by the sole selection of a suitable M(q) matrix. This will be 
explicitly used in Section 4. 

The form of the covariance function can be further stud- 
ied. The eigenvalues of £ read = a + f3e~s~ and can 
be rewritten as X k = e T k e T k , k = 1, . . . [d/2\ (where 
[z\ stands for the integer part of z). For ease of notations, let 
{C M )k = J2i M i-k> {Lu)k = E c M k,c and let T denote the 
(non-normalized) discrete Fourier transform. Then, detailed 
calculations enabled us to show that, for any q, 



e[x«x«; 



E[X«]E[X?} = 



|d/2j 

fc=i ^ 

(12) 

where = 1 if 2k — d and 2 otherwise. This result shows 
that Tfc and are characteristic dependence lengths and pe- 
riodicities that depend on the joint choices of d and a. More 
precisely, from the definition of r k = — 1/ In |Afc|, we can 

show that Tfc « Q( 3^o [a/3 (l — cos 2 -^)] . Enlarging the 
dimension <i of the matrices Rd increases both the number of 
distinct dependence lengths as well as the ratio of the smallest 
to the largest such characteristic lengths, which varies asymp- 
totically as (d/2n) 2 /2. 

Therefore, choosing d, a, hence £, and the sequence of 
Vij enables to select the marginal and joint distributions ac- 
cording to given targets. Section 4 will illustrate the potential 
of the method. 



3. SYNTHESIS: HIDDEN MARKOV CHAIN 

While the compact form of P(X) in Eq. (|2| constitutes a gen- 
eral framework for the analytical derivation of numerous sta- 
tistical properties of X< it provides few insights with respect 
to its numerical synthesis. To address this issue, a reformu- 
lation of Eq. Q into the form of a Hidden Markov Model is 
now devised. 




4. ILLUSTRATIONS 



a a 



Fig. 1. Transition graph. For d = 6 and £ = aid + PJd 

First, extending to product of N matrices the fact that 
the entries of the matrix product (ABC) reads (ABC)ij — 
Sfe i a i,kbk,iCij, we have been able to recast Eq. (|2ji into: 
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It is straightforward to verify that J^r = 1> hence 

«(r) can be interpreted as the probability function of T, 



Moreover, Eq. (13i shows that ¥(X) can be read as a 



«(r)-weighted mixture of laws, each defined as the pvod- 

Second, from Eq. (14) , using the special form of A and £, 
it can be shown that: 



P(r fc =j|r = 7o,---r fe _i = 7fc-i) = 

7i • • • ^7t-i.j ^i.rfc+i ■ ■ ■ £rjy-i,rjy 



(15) 
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and hence that T is a homogeneous c?-state Markov chain, 
with transition probability matrix: 



p(r fc+1 =j|r fc = *) = £i 



1,3- 



(16) 



Hence, £ can be recast as the transition matrix of the Markov 
chain T, as illustrated in Fig.[T] 

Third, the final step required for numerical synthesis is to 



derive the initial distribution for To Eq. ( [T4[ i enabled us to 

lifo 

p(r - i) 



(17) 



show that it should follow a uniform distribution: 

1 

d' 

Therefore, the time series X_ k , as defined from Eq. ([TJ, 
can be read and interpreted as a Hidden Markov Model, with 
2 hidden states: the current state T k and the previous state 

rVi. 



Combining Eqs. ( 16 1 and ( 17 1, a synthesis algorithm can 



be sketched as follows: 

Step 1 : Initialization: Use Eq. ( 17 1 to generate the state To. 
Step 2 : Iteration on k: 

i) Choose at random state ]?/., using Eq. < p~6| > , 

ii) Generate X k according to T , r k _ 1 .r k ■ 




Fig. 2. Numerical Synthesis. Two different times series 
with the same marginal distribution (mixture of two Gaus- 
sians, o\ — 0.5, o-i = 2), the same covariance functions 
(chosen as S functions) but different covariance functions for 
their squares, hence different prescribed joint distributions 
(a = 0.98). Left side X_, right side Y_, First line, one re- 
alization of the time series. Then, from top to bottom, es- 
timated (solid black lines) and theoretical (dashed colored 
lines) marginals, correlation functions and correlation func- 
tions for the squared time series. 

To illustrate the potential of the proposed time series the- 
oretical construction and synthesis procedure, a pedagogical 
example is proposed and the construction of the targeted sta- 
tistical properties is devised: It consists of a pair (X_, Y) of 
processes sharing the same marginal distributions (a mixture 
of two Gaussian laws), the same autocovariance functions (a 
S function, i.e., no correlation), but different joint distribu- 
tions, hence different higher order statistic dependences. In 
this example, the dependence of the 4— th order statistics will 
be prescribed. 

We select d = 6 and £ = aid + @Jd- For the sake 
of simplicity, the univariate (or marginal) distributions is 
set to be a mixture of two Gaussian distributions: p(x) — 
\ CM),cri ( x ) + M),tT2 i x ))- F° r that case, it is natural to envis- 
age that the matrix V consists only of Gaussian distributions. 



Note however that this is not the sole possibility. This restric- 
tion implies that A/o, ffl and No,cr 2 must appear exactly three 
times both in the principal and upper circular diagonals of V . 
In other words, selecting the univariate distribution fixes the 
number of occurrences of each law in matrix V, but not their 
position in the matrix. 

Moreover, to make the example convincing, we chose 
the covariance functions of both X_ and Y_ to be <5-functions. 
These two time series will hence have no autocorrelation but 
(higher-order) dependencies. To achieve this, it is sufficient 
that matrix M(l) is a zero matrix (cf. Eq. (111). In our 



example, this is automatically obtained due to the choice of 
zero-mean distributions: A/"o, CTl andA/o l(T2 . 

These two time series have hence the same marginal dis- 
tributions and covariance functions. They are yet different by 
construction as we now impose that they have different joint 
distributions. Taking into account the circularity of the tran- 
sition graph in Fig. [T] there exist 80 distinct choices of V, 
consisting of exactly 6 entries set to No, ai an d 6 entries set 
to A/o,cr 2 > leading to different joint distributions. Making use 
of Eq. (12 1 and of the explicit form of the eigenvalues of £ 



for d = 6 enables us to show that E [X$X?] - E [Xg] E [Xf], 
for any q, stems from the superimposition of three distinct 
exponential terms whose characteristic lengths, we derived 



asymptotically: r% 



13^0 
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and T3 



J_ 
20- 



The 



actual choice for the entries of V fixes the complex valued 
coefficients, associated to each of these correlation lengths 



(cf. Eq. 12 1. As demonstrative examples, we selected for X: 
Vfj = A/o.o-i if i even, and Vfj = A/o )<Ta if i odd. Applying 
Eq. (jT2j to that choice, shows that the sole T3, i.e., the shortest 
correlation length, appears in the auto-covariance of X 2 (the 
complex-valued coefficients associated to the two other cor- 
relation lengths are forced to by the specific choice made 

_ (e>"i-cr 2 ) 2 f„ a\t 

= No. 



here): E [X 2 X 2 ] - E [Xq] E [X 2 ] = (a - /?)* 

* - Kr ~ ,if 1 < i < 3 and PL = 



Conversely, choosing V\ 



A/"o )<7a if 4 < i < 6 leads to E [X$X 2 ] - E [Xg] E [X 2 ] = 

{ai ~ d P )2 [{a - +Wt{{2 - p - V3i/3)\\- In that 
case, both t\ and T3 contribute to the autocovariance of Y 2 . 
However, t\ ps At% is dominant at large t and hence consti- 
tutes the leading term. Using the synthesis procedure devised 
in Section 3 for this pair of examples yields Fig. [2j show- 
ing for times series X_ (left column) and Y (right column), 
a particular realization, the estimated and targeted univariate 
distributions, covariance functions and covariance functions 
for the squared time series (from top to bottom). It clearly 
shows that X_ and Y have the same marginal and covariance 
but different joint distributions (as targeted). 

Using the same construction procedure, other pairs of ex- 
amples with the same marginals, same (non 5-) autocovari- 
ance functions but different joint distributions could as easily 
be devised. Mixture of Gaussians are used here by conve- 
nience, but mixtures of any other valid distributions could just 
as easily be reached. 



5. CONCLUSION AND PERSPECTIVES 

Inspired from Statistical Physics models, a general frame- 
work enabling us to define the joint distributions of a random 
vector X_ has been described. It has then been specified to 
the definition of stationary time series, with control of their 
joint distributions and explicit derivation of numerous of their 
statistical properties. A remapping onto a Hidden Markov 
Model enabled us to devise an efficient synthesis procedure, 
available upon request (in Matlab). Constructive examples 
aiming at showing the potential of the tools were proposed. 
This general framework will be further explored by depart- 
ing from the restrictive choices made here for the matrices 
A and £. This should notably enable us to define (and syn- 
thesize numerically) vectors of dependent variables with 
different marginal distributions and intricate dependencies. 
Notably, the intriguing and promising special cases of non- 
diagonalisable matrices £ are expected to offer a much larger 
versatility in the form of dependencies that can actually be 
reached. This is under current investigations. 

J.-Y. Tourneret and N. Dobigeon are gratefully acknowl- 
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